Partial zarr download/upload (stage: Design + Implementation) by yarikoptic · Pull Request #1816 · dandi/dandi-cli

yarikoptic · 2026-03-02T21:37:30Z

Summary

Design and implementation for partial zarr download and upload support, addressing #1462, #1474, and related archive issues.

The PR covers five areas:

--zarr TYPE:PATTERN filtering for dandi download — glob, path, and regex filters for selecting entries within zarr assets, with a metadata alias for common zarr metadata files
URL parsing with zarr boundary detection — AssetZarrEntryURL to handle URLs like dandi://dandi/000108/.../file.ome.zarr/0/0/0
--zarr-mode {full, patch} for dandi upload — patch mode uploads changed files without deleting remote files absent locally
Checksums and manifests — documents that per-directory checksums are computed hierarchically by the zarr_checksum library but are NOT persisted (only the root digest is stored in the DB); legacy .checksum files exist on S3 at zarr-checksums/ for ~72% of older zarrs but are orphaned since Dec 2022
dandi ls for zarr contents — listing files within a zarr asset when URL points into a zarr

Key findings from investigation

The zarr_checksum algorithm IS hierarchical (Merkle tree, bottom-up via ZarrChecksumTree)
The archive's ingest_zarr_archive task computes checksums entirely in memory and stores only the root digest
Per-directory .checksum files on S3 (zarr-checksums/ prefix) were written by ZarrChecksumFileUpdater, removed in dandi-archive PRs Testing started to fail due to an error in parsing (?) + new deprecationwarning #1390/[FEAT] dandi fsspec filesystem #1395 (Dec 2022). Legacy files remain for older zarrs but no API exposes them
Subtree checksum verification is not possible today without recomputation from file ETags

Implementation

New files

dandi/zarr_filter.py — ZarrFilter dataclass with glob/path/regex matching, parse_zarr_filter(), make_zarr_entry_filter(), ZARR_FILTER_ALIASES (includes metadata alias)
dandi/tests/test_zarr_filter.py — 52 unit tests covering all filter types, parsing, aliases, edge cases, invalid regex validation

Modified files

dandi/dandiarchive.py — split_zarr_location(), AssetZarrEntryURL class, updated parse_dandi_url() to detect zarr boundaries
dandi/download.py — zarr_filters parameter on download(), filter threading through Downloader and _download_zarr(), skip deletion/checksum when filter active
dandi/files/zarr.py — zarr_mode: Literal["full", "patch"] on iter_upload(), patch mode skips remote file deletion and client-side checksum verification
dandi/upload.py — zarr_mode parameter on upload(), conditional pass-through for ZarrAsset
dandi/cli/cmd_download.py — --zarr click option (multiple, OR logic)
dandi/cli/cmd_upload.py — --zarr-mode click option
dandi/cli/cmd_ls.py — list zarr entries when URL is AssetZarrEntryURL
dandi/cli/tests/test_download.py — updated mock expectations for new zarr_filters parameter

Integration tests

dandi/tests/test_download.py — 6 tests (glob filter, metadata alias, no-delete, path filter, nonexistent filter, sync conflict)
dandi/tests/test_upload.py — 3 tests (patch no-delete, full delete, patch updates)
dandi/tests/test_dandiarchive.py — 3 URL parsing cases + 8 split_zarr_location cases

Review checklist

Please review the design at doc/design/partial-zarr.md and comment on:

--zarr TYPE:PATTERN syntax — is the filter approach right? Are glob/path/regex the right types?
metadata alias expansion — does glob:**/.z* + glob:**/zarr.json + glob:**/.zmetadata cover all cases?
--zarr-mode patch semantics — is "upload without deleting" the right default for patch? ~~Should subtree cleanup happen?~~
URL parsing — is AssetZarrEntryURL with zarr boundary detection the right approach?
Checksum strategy — relying on per-file ETags for partial ops, deferring subtree checksums to future manifests
Open questions in the doc (OR composition chosen, --sync conflict raises error, server-side glob deferred)
Should legacy zarr-checksums/ files on S3 be cleaned up as part of this or separately?

TODO

🤖 Generated with Claude Code

codecov · 2026-03-02T21:45:00Z

Codecov Report

❌ Patch coverage is 91.64491% with 32 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.57%. Comparing base (5f03d9b) to head (e32fb86).
⚠️ Report is 131 commits behind head on master.

Files with missing lines	Patch %	Lines
dandi/dandiarchive.py	64.28%	10 Missing ⚠️
dandi/download.py	86.44%	8 Missing ⚠️
dandi/upload.py	37.50%	5 Missing ⚠️
dandi/zarr_filter.py	95.31%	3 Missing ⚠️
dandi/cli/cmd_ls.py	60.00%	2 Missing ⚠️
dandi/files/zarr.py	90.00%	2 Missing ⚠️
dandi/tests/test_download.py	96.15%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1816      +/-   ##
==========================================
+ Coverage   75.12%   75.57%   +0.44%     
==========================================
  Files          84       86       +2     
  Lines       11930    12230     +300     
==========================================
+ Hits         8963     9243     +280     
- Misses       2967     2987      +20

Flag	Coverage Δ
unittests	`75.57% <91.64%> (+0.44%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Covers five areas: - --zarr TYPE:PATTERN filtering for download (glob, path, regex) - URL parsing with zarr boundary detection (AssetZarrEntryURL) - --zarr-mode {full, patch} for upload - Checksums and manifests (per-directory checksums are NOT persisted on the archive; legacy .checksum files exist on S3 under zarr-checksums/ for ~72% of older zarrs but are orphaned) - dandi ls for zarr contents Related: #1462, #1474 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add --zarr CLI option for download to filter entries within zarr assets (glob/path/regex patterns with predefined 'metadata' alias), and --zarr-mode option for upload to support 'patch' mode (upload/update without deleting remote-only files). Key changes: - New dandi/zarr_filter.py: filter parsing, matching, and aliases - URL parsing: AssetZarrEntryURL for URLs pointing into zarr assets - Download pipeline: thread zarr_entry_filter through Downloader and _download_zarr, skip deletion and checksum when filter active - Upload pipeline: zarr_mode='patch' skips remote file deletion and client-side checksum verification - dandi ls: list zarr entries when URL points into a zarr Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Pre-compile regex patterns at ZarrFilter construction time, catching invalid patterns early instead of on every matches() call (B1) - Remove redundant get_asset_download_path override in AssetZarrEntryURL that was identical to the inherited SingleAssetURL method (H1) - Use Literal["full", "patch"] for zarr_mode parameter instead of bare str to prevent silent misbehavior on invalid values (H2) - Collapse consecutive ** glob segments to avoid exponential backtracking in _glob_match_parts (H3) - Simplify split_zarr_location to use str.split instead of PurePosixPath (M1) - Add explanatory comment for type: ignore in parse_zarr_filter (M2) - Yield {"status": "done"} when zarr filter matches zero entries (M4) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

click.Choice returns str at runtime, but upload() expects Literal["full", "patch"]. Add typing cast at the call site. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Follow existing codebase pattern (UploadExisting, DownloadExisting, etc.) to define ZarrMode as a str Enum in upload.py. Eliminates the duplicated Literal["full", "patch"] across three files and the cast() workaround in cmd_upload.py. Uses TYPE_CHECKING guard in files/zarr.py to avoid circular import (files/zarr.py -> upload.py -> .files). Co-Authored-By: Claude Code 2.1.63 / Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Code 2.1.63 / Claude Opus 4.6 <noreply@anthropic.com>

kabilar · 2026-06-15T18:57:49Z

+## Primary Use Case
+
+From @kabilar (#1462):
+
+```bash
+# 1. Download just the metadata files from a zarr
+dandi download --zarr glob:'**/.z*' --zarr glob:'**/zarr.json' \
+    dandi://dandi/001289/rawdata/.../PC.ome.zarr
+
+# 2. Edit .zattrs locally
+vim PC.ome.zarr/.zattrs
+
+# 3. Upload changes without deleting remote data chunks
+dandi upload --zarr-mode patch
+```


Thanks @yarikoptic. An additional use case would be adding shards to an existing Zarr. For example, we have 355+ TB from a subject that were are stitching together into a single Zarr. We don't have enough space locally so will need to upload a shard, delete the local copy, and repeat the process.

cc @satra @ayendiki

kabilar · 2026-06-15T20:29:58Z

+See separate implementation task.  Summary: allow `dandi ls` to list files
+within a Zarr asset when given a Zarr URL, using `asset.iterfiles(prefix=...)`.
+Reuses `AssetZarrEntryURL` from Part 2.


Suggested change

See separate implementation task. Summary: allow `dandi ls` to list files

within a Zarr asset when given a Zarr URL, using `asset.iterfiles(prefix=...)`.

Reuses `AssetZarrEntryURL` from Part 2.

This will be implemented as a separate pull request. Summary: allow `dandi ls` to list files

within a Zarr asset when given a Zarr URL, using `asset.iterfiles(prefix=...)`.

Reuses `AssetZarrEntryURL` from Part 2.

kabilar · 2026-06-15T20:34:16Z

+1. **`--zarr` AND vs OR**: Multiple `--zarr` options use OR.  Should we support
+   AND composition (e.g., `--zarr path:0/1 --zarr-and glob:'**/.zattrs'`)?
+   For now, OR suffices; AND can be added later.


Just having the OR operator is fine with me.

kabilar · 2026-06-15T20:40:30Z

+3. **Interaction with `--sync`**: When `dandi download --zarr ... --sync` is
+   used, should `--sync` only delete local zarr entries that match the filter
+   but aren't remote?  Or should `--sync` be disallowed with `--zarr`?
+   Recommendation: disallow `--sync` with `--zarr` initially, as the semantics
+   are ambiguous.


As mentioned, since the semantics could be confusing, I would suggest disallowing --sync with --zarr for dandi download.

kabilar · 2026-06-15T20:47:44Z

+2. **Server-side glob for zarr entries**: Currently the server only supports
+   `prefix` filtering on zarr entries (not glob).  Glob filtering therefore
+   happens client-side after fetching entries.  For large zarrs with millions of
+   entries, this could be slow.  A future server API enhancement for
+   entry-level glob would help.  For now, `path:` filters should be preferred
+   for large zarrs to minimize data transfer.


This works. We are using Zarr v3 so glob filtering on the client-side shouldn't be an issue.

kabilar · 2026-06-15T20:49:02Z

+| `dandi/files/zarr.py` | `iter_upload()` patch mode support |
+| `dandi/cli/cmd_ls.py` | Zarr contents listing (separate PR) |
+
+## Open Questions


I agree with all of the proposed solutions for the open questions.

kabilar · 2026-06-15T20:52:53Z

Thanks @yarikoptic. This design looks good and works for our two use cases: 1. updating Zarr metadata, and 2. adding chunks/shards to an existing Zarr.

yarikoptic marked this pull request as draft March 2, 2026 21:37

yarikoptic requested review from kabilar and satra March 2, 2026 21:38

yarikoptic changed the title ~~Design: partial zarr download/upload~~ Partial zarr download/upload (stage: Design) Mar 2, 2026

yarikoptic mentioned this pull request Mar 2, 2026

Add versioned Zarr design doc dandi/dandi-archive#2702

Draft

yarikoptic added enhancement New feature or request minor Increment the minor version when merged labels Mar 3, 2026

yarikoptic and others added 3 commits March 6, 2026 08:10

yarikoptic force-pushed the partial-zarr branch from dd19691 to 16b25ec Compare March 9, 2026 18:09

yarikoptic changed the title ~~Partial zarr download/upload (stage: Design)~~ Partial zarr download/upload (stage: Design + Implementation) Mar 9, 2026

yarikoptic and others added 3 commits March 11, 2026 11:34

Fix mypy error: cast click str to Literal for zarr_mode

a9d61f4

click.Choice returns str at runtime, but upload() expects Literal["full", "patch"]. Add typing cast at the call site. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Move zarr_filter imports to module level in download.py

e32fb86

Co-Authored-By: Claude Code 2.1.63 / Claude Opus 4.6 <noreply@anthropic.com>

kabilar reviewed Jun 15, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Partial zarr download/upload (stage: Design + Implementation)#1816

Partial zarr download/upload (stage: Design + Implementation)#1816
yarikoptic wants to merge 6 commits into
masterfrom
partial-zarr

yarikoptic commented Mar 2, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Mar 2, 2026 •

edited

Loading

Uh oh!

kabilar Jun 15, 2026

Uh oh!

kabilar Jun 15, 2026

Uh oh!

kabilar Jun 15, 2026

Uh oh!

kabilar Jun 15, 2026

Uh oh!

kabilar Jun 15, 2026

Uh oh!

kabilar Jun 15, 2026

Uh oh!

kabilar Jun 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

yarikoptic commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key findings from investigation

Implementation

New files

Modified files

Integration tests

Review checklist

TODO

Uh oh!

codecov Bot commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

kabilar Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

kabilar Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

kabilar Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

kabilar Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

kabilar Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

kabilar Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

kabilar Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yarikoptic commented Mar 2, 2026 •

edited

Loading

codecov Bot commented Mar 2, 2026 •

edited

Loading